Index Compression through Document Reordering

An important concern in the design of search engines is the construction of an inv erted index. An inv erted index, also called a concordance, contains a list of documents (or posting list) for every possible search term. These posting lists are usually compressed with di erence coding. Di erence coding yields the best compression when the lists to be coded hav e high locality. Coding methods hav e been designed to speci cally take advantage of locality in in v ertedindices. Here, we describe an algorithm to permute the document n umbers so as to create locality in an inv erted index. This is done b yclustering the documents. Our algorithm, when applied to the TREC ad hoc database (disks 4 and 5), improv es the performance of the best di erence coding algorithm we found by fourteen percent. The improvement increases as the size of the index increases, so we expect that greater improv ements would be possible on larger datasets.